珍云恒星: 2023.11 truncation参数默认取值为False("do_not_truncate") 参考官网链接 https://huggingface.co/docs/transformers/v4.35.2/en/main_classes/tokenizer#transformers.PreTrainedTokenizer truncation (bool, str or TruncationStrategy, optional, defaults to False) — Activates and controls truncation. Accepts the following values: True or 'longest_first': ... 'only_first': ... 'only_second': ... False or 'do_not_truncate' (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size).

m0_63292668: indexed_tokens = tokenizer.encode("Hi, my name is Mike.") print(indexed_tokens) # [17250, 11, 616, 1438, 318, 4995, 13] a = tokenizer.tokenize('Hi, my name is Mike.') print(a) # ['Hi', ',', 'Ġmy', 'Ġname', 'Ġis', 'ĠMike', '.'] print(tokenizer.convert_tokens_to_ids(a)) # [17250, 11, 616, 1438, 318, 4995, 13] --------------------------------------------------------- 2023年现在是不是变了, 直接用encode和两步走的结果都一样

。七十二。: 你好，请问我要将服务器上的代码使用vs code把他从服务器上传到gitlab，这个ssh-key是复制服务器上的吗？

frr0717: 🈶个人风格的参数类，比如？我这方面经验比较少，想请教下前辈

征途黯然.: 写得不错呀博主，继续关注!